Augmented reality navigational overlay

ABSTRACT

A system and method may generate a more realistic augmented reality (AR) overlay by generating a segmentation image and blending it with one or more other images. The system may generate a segmentation image based on an input image. The segmentation image may be blended with an AR path overlay image to generate an object-masked AR path overlay image. The object-masked AR path overlay image may be blended with the input image to generate an output image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part application of U.S.application Ser. No. 16/569,650, filed Sep. 12, 2019, which is acontinuation of U.S. application Ser. No. 16/255,825, filed Jan. 24,2019, which claims the benefit of U.S. Provisional Patent ApplicationNo. 62/737,563, filed Sep. 27, 2018, which are all hereby incorporatedby reference in their entirety.

TECHNICAL FIELD

The present disclosure generally relates to real-time navigation systemsand, more particularly, to augmented reality navigation systems andmethods.

BACKGROUND

Computer implemented real-time navigation systems are commonly providedin vehicles and on mobile devices. These navigation systems useglobal-positioning-system (GPS) coordinates of the system or the devicein which the system is implemented to display, in real time, the currentlocation of the system on a map. These navigation systems can providereal-time, or turn-by-turn directions from the current location to adestination location for vehicle, bicycle, walking, or publictransportation. These navigation systems often display acomputer-generated depiction of the roads and sidewalks, along with anindicator of the current location, and an indicator of the nextdirection along a route. The indicator of the next direction can be anarrow that extends from the current location in the direction in whichthe user should proceed, such as an indicator of a left turn, a rightturn, a roadway exit, a roadway entrance, or an indicator to proceedalong the current direction.

However, these navigation systems typically use stored maps that may notinclude new or mobile objects in the user's path.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

The accompanying drawings, which are included to provide furtherunderstanding and are incorporated in and constitute a part of thisspecification, illustrate disclosed embodiments and together with thedescription serve to explain the principles of the disclosedembodiments. In the drawings:

FIG. 1A is a schematic illustration of augmented reality navigationoperations according to certain aspects of the present disclosure.

FIGS. 1B-1F further illustrate the schematic illustration of theaugmented reality navigation operations according to FIG. 1A.

FIG. 1G illustrates an alternate schematic illustration of the augmentedreality navigation operations according to FIG. 1A.

FIGS. 2A-B illustrate a flow chart of example processes for augmentedreality navigation in accordance with various aspects of the subjecttechnology.

FIG. 3 illustrates an electronic system with which one or moreimplementations of the subject technology may be implemented.

FIG. 4A illustrates a schematic illustration of the augmented realitynavigation operations according to certain aspects of the presentdisclosure.

FIG. 4B illustrates another schematic illustration of the augmentedreality navigation operations according to certain aspects of thepresent disclosure.

FIG. 5 illustrates a flow chart of an example process for augmentedreality navigation overlay in accordance with various aspects of thepresent disclosure.

DETAILED DESCRIPTION

The detailed description set forth below describes variousconfigurations of the subject technology and is not intended torepresent the only configurations in which the subject technology may bepracticed. The detailed description includes specific details for thepurpose of providing a thorough understanding of the subject technology.Accordingly, dimensions may be provided in regard to certain aspects asnon-limiting examples. However, it will be apparent to those skilled inthe art that the subject technology may be practiced without thesespecific details. In some instances, well-known structures andcomponents are shown in block diagram form in order to avoid obscuringthe concepts of the subject technology.

It is to be understood that the present disclosure includes examples ofthe subject technology and does not limit the scope of the appendedclaims. Various aspects of the subject technology will now be disclosedaccording to particular but non-limiting examples. Various embodimentsdescribed in the present disclosure may be carried out in different waysand variations, and in accordance with a desired application orimplementation.

In the following detailed description, numerous specific details are setforth to provide a full understanding of the present disclosure. It willbe apparent, however, to one ordinarily skilled in the art thatembodiments of the present disclosure may be practiced without some ofthe specific details. In other instances, well-known structures andtechniques have not been shown in detail so as not to obscure thedisclosure.

As an improvement to existing computer implemented real-time navigationsystems in which guidance indicators are displayed together with alocation indicator and a map, augmented reality navigation systems andmethods are provided.

Augmented reality navigation systems and methods provide enhancedguidance indicators in combination with a real-time view of a real-worldguidance route. The real-time view of the real-world guidance route mayinclude a direct view to the route by a user of the system or may be animage of the environment around the system that includes the route.

The enhanced guidance indicator is overlaid on the direct view or imageof the real-time view to indicate to the user of the system, where toproceed in the real-world environment to follow the real-world guidanceroute. The enhanced guidance indicator provides an improvement over theguidance arrows of conventional navigation systems that are displayedover a map, at least in that the enhanced guidance indicator isgenerated and displayed responsive to real-time images of the upcomingroute at the current time and at the current location of the system.This reduces or eliminates the need for the user to direct attentionaway from driving or otherwise proceeding along the guided route todetermine how the displayed map correlates to the real-world scene inthe user's view.

The route may include a roadway (e.g., a street, a highway, a freeway, atollway, a parking lot, or the like) or other pathway (e.g., a sidewalk,a walking path, a bike path, a stairway, an escalator, or any otherindoor or outdoor real-world environment) through or along which thenavigation system is guiding the user. As examples, when the augmentedreality (AR) navigation system is being used in a car or otherroad-based vehicle, the route may include a path along the roadway onwhich the vehicle is currently located, and the real-time view includesthe portion of the roadway over which the guidance is directing theuser. The enhanced guidance indicator is overlaid on this real-time viewto display the AR navigation guidance to the user.

Because the roadways and other pathways along which users are guided aretypically unchanging from day to day, week to week, or over periods ofyears or decades, generating guidance indicators for routes on theseroadways or pathways benefits from prior knowledge of the location,size, width, and other aspects of the roadways or pathways. Accordingly,where a portion of a particular street curves to the right, an enhancedguidance indicator that is overlaid on an image of that portion of thatparticular street can correspondingly curve to indicate the AR guidedroute. Moreover, where a left turn onto a new street is to be instructedby the system, the enhanced guidance indicator that is overlaid on animage of the intersection between the current street and the new streetcan be bent to indicate the AR guided turn. These enhanced guidanceindicators can be sized, shaped, and located to conform to thecorresponding roadway features based on real-time images of the roadwaysand/or a priori knowledge of the roadway features.

However, at any given time, other objects such as other vehicles,bicyclists, pedestrians, or debris that are not known a priori to thesystem can be located in the route, often at locations over which theenhanced guidance indicators are to be overlaid on the scene (e.g., onan image or a direct view of the scene).

In accordance with various aspects of the subject disclosure, enhancedguidance indicators are displayed that adapt, in real time, to thechanging objects in the route so that the objects are visible, and theguidance indicators appear to the user to be overlaid on the roadway orpathway, rather than over objects on the roadway or pathway. In thisway, an improved AR navigation system is disclosed that displays a morerealistic AR overlay, appearing to lie underneath pathway objects ratherthan blended with pathway objects, which may enhance the safety ofdisplaying navigational guidance to a user, particularly when the useris operating a vehicle. As described in further detail hereinafter, inorder to display AR guidance in which roadway or pathway objects appearon top of a guidance indicator overlay, the guidance overlay may haveportions that overlap the objects dimmed, or removed, to create theeffect of that object occluding the guidance overlay.

FIGS. 1A-F illustrate an example in which AR navigation guidance isdisplayed in a scenario in which an object on a roadway occludes aguidance indicator that indicates the recommended path for a user. Inthe example of FIG. 1A, an AR navigation system 100 a receives,generates, and displays images for AR navigation guidance. AR navigationsystem 100 a receives an input image such as input image 100 b. Theinput image 100 b may be captured by a camera. The camera may be acamera of a mobile device such as a smart phone in which the ARnavigation system 100 a is implemented or the camera may be a camera ofa vehicle incorporating the AR navigation system 100 a orcommunicatively coupled to the AR navigation system 100 a (as examples).

As shown in FIG. 1A and FIG. 1B, input image 100 b approximates thecurrent view of the real-world scene in the user's view (e.g., throughthe windshield of a vehicle while driving). In this example, input image100 b includes roadway 102, a vehicle 104 driving on the same roadway102 as that of the user, at least one other vehicle captured in theinput image 100 b, and other objects 106 such as trees, sidewalks, etc.that are not on roadway 102.

As indicated in FIG. 1A, the AR navigation system 100 a may generate asegmentation image 100 c based on input image 100 b, further shown inFIG. 1C. As shown in FIGS. 1A and 1C, segmentation image 100 c may be abinary object mask image having the same number of pixels and samedimensions as that of input image 100 b. The binary object mask imageincludes having pixels 110 set to a first common value (e.g., one) whereroadway objects such as vehicle 104 are located in input image 100 b,and having pixels 112 set to another common value (e.g., zero)elsewhere. In this example, all of the pixels of the segmentation image100 c will either be pixel 110 with the first common value or pixel 112with the another common value different from that of pixel 110.Segmentation image 100 c may be generated by detecting (e.g., usingartificial intelligence operations such as deep learning semanticsegmentation operations) the roadway, roadway objects such as vehicle104 on the roadway, and other objects such as objects 106 in input image100 b. The AR navigation system 100 a can use the artificialintelligence operations to detect and delineate among the differentobject and set pixels in the segmentation image that correspondspatially to the roadway objects in input image 100 b to the firstcommon value. In one example, the pixels derived from input image 100 bthat are assigned the second common value can be that of any backgroundpixels not on the road, road pixels, or pixels that indicate areasadjacent to the road. The remaining pixels not assigned with the secondcommon value can be assigned with the first common value. In oneexample, other objects on the roadway other than vehicle 104 can bedetected such as a pedestrian, a bicycle, a pedestrian, or any othermoving object relative to the road and be assigned the first commonvalue in the segmentation image 100 c.

In one example, the segmentation image can be generated by usingartificial intelligence operations semantic segmentation operations. Inthis example, the AR navigation system 100 a can semantically identifyand segment the image, such as input image 100 b, into regions. Theregions can be classified or based on portions or one or more pixels ofthe image that are identified as moving objects or roadway objects, orboth.

In one example, the semantic segmentation is performed based on adetection of changes in pixels to indicate a moving object. In anembodiment, pixels that do not change may be determined to bebackground. Pixels that do change may be determined to be movingobjects. In another example, the frame of reference of the vehicle inwhich the camera is located is moving so pixels of background objectsmay move. However, motion of the vehicle may be modeled, such as by amotion equation of the vehicle accounting for its velocity andacceleration, and pixels changing according to the motion equation ofthe vehicle may be determined to be background and pixels that do notchange according to the motion equation of the vehicle may be determinedto be other moving objects.

In one example, an object detection algorithm may be used to assist withsemantic segmentation. Object detection may be performed in someexamples by a neural network, such as a convolutional neural network ora multi-layer neural network in which at least one layer is aconvolutional neural network. The object detection algorithm may acceptas input an image or portion of an image and output a probabilitydistribution that models the probability that the image or image portioncontains an object of a specific type, such as a car, bicycle, orpedestrian. The highest probability outcome may be selected as theoutput object classification. The object detection algorithm may bebased on machine learning and trained on training examples of knownobjects and associated training labels that identify the known objects.The object detection algorithm may identify moving objects in the scenewith bounding regions and may also output feature tensors representingfeatures used to identify the objects. Segmentation may use the boundingregions and feature tensors to classify individual pixels based onwhether they belong to a particular object or the background. In oneexample, the object detection and segmentation steps may be performed bya single machine learning model, such as a neural network or amulti-layer neural network with one or more convolutional layers.

In one example, a deep, multi-layer neural network with one or moreconvolutional neural network layers is used to perform semanticsegmentation. The deep neural network may accept as input an image orportion of an image and output a probability distribution for theclassification of each pixel, for example, to a particular type ofobject, such as a car, bicycle, or pedestrian, or to background. Thehighest probability outcome may be selected as the pixel'sclassification. The deep neural network may be trained on trainingexamples of images with pixel-level classifications identifying objectsand background.

In another example, the object mask image can be generated based methodsother than the use of artificial intelligence.

As indicated in FIGS. 1A and 1D, in addition to (e.g., in parallel with)the generation of segmentation image 100 c, an AR path overlay image 100d is also generated by the AR navigation system 100 a. As shown, AR pathoverlay image 100 d includes a guidance indicator 116, to indicate thedirection in which the user should guide the vehicle to proceed alongthe guidance route. In one example, the guidance indicator 116 cancorrespond to a specific road lane that the user or the AR navigationsystem 100 a is currently on. In one example, the guidance indicator cancorrespond to multiple lanes or portions of the road. The guidanceindicator 116 can be generated by using GPS data for the route that theuser has set or has been set for the user by the AR navigation system100 a.

In one example, AR navigation system 100 a first generates thesegmentation image 100 c and then subsequently generates the AR pathoverlay image 100 d. In another example, the AR navigation system 100 afirst generates the AR path overlay image 100 d and then subsequentlygenerates the segmentation image 100 c. In another example, the ARnavigation system 100 a concurrently generates the segmentation image100 c and the AR path overlay image 100 d.

As indicated in FIGS. 1A and 1E, segmentation image 100 c and AR pathoverlay image 100 d may be blended by a combiner 118. Combiner 118 mayblend segmentation image 100 c and AR path overlay image 100 d. Oncecombined, the AR navigation system 100 a can identify the pixels in thesegmentation image 100 c having the first value (e.g., one) that overlapwith the pixels that form guidance indicator 116 of the AR path overlayimage 100 d. The AR navigation system 100 a can then modify thecorresponding pixels of the guidance indicator 116 in AR path overlayimage 100 d (e.g., by multiplying the corresponding pixels with a scalarthat is than one to dim that portion of AR path overlay image 100 d orby setting the corresponding pixels to zero to effectively remove thepixels). The result will cause a dimming effect of at least a portion ofthe guidance indicator 116 of AR path overlay image 100 d. For example,a scalar of 0.8 applied to the portion of pixels of guidance indicator116 that overlaps with any first pixel 110 of segmentation image 100 cwill result in that portion of pixels of guidance indicator of 116 havea 20% reduction in brightness. In one example, the modification of thepixels of the guidance indicator by the scalar can be that of opacity.

As indicated in FIGS. 1A and 1E, combiner 118 generates an object-maskedAR path overlay image 100 e that includes a modified guidance indicator117 having a blended portion 122 modified by combiner 118 based onsegmentation image 100 c. In this example, the blended portion 122 willinclude the portion of guidance indicator 116 that overlaps with anyfirst pixel 110 of segmentation image 100 c and result in a brightnessor opacity different from that of the brightness or opacity of the restof the guidance indicator 116. The combined blended portion 122 and theremaining unmodified portion of guidance indicator 116 will form themodified guidance indicator 117.

As indicated in FIG. 1A and 1F, the AR navigation system 100 a may thencombine (e.g., add) object-masked AR path overlay image 100 e with inputimage 100 b, by a combiner 119, to display an output image 100 f. Asshown in FIGS. 1A and 1F, output image 100 f includes input image 100 band modified guidance indicator 117. Due to the difference in brightnessor opacity, or both of blended portion 122 or modified guidanceindicator 117, the AR path overlay in output image 100 f appears to fallunderneath car 104.

For comparison, as illustrated in FIG. 1G, image 100 g is also shown, inwhich input image 100 b is combined with the unmodified AR path overlayimage 100 d. As shown, in image 100 g, guidance indicator 116 appears tofall on top of the vehicle 104, which may be confusing or otherwiseundesirable to the user, particularly if multiple roadway objects suchas other vehicles, pedestrians, bicyclists, etc. are present andoccludes the AR path overlay image displayed to the user.

It should also be appreciated that, output image 100 f may be displayedon the user's mobile phone or on a display of a dedicated navigationdevice, or object-masked AR path overlay image 100 e may be projectedonto a direct view of the scene by the user (e.g., using a head-updisplay or by projecting modified guidance indicator 117 onto thewindshield of the vehicle in which the user is located). Although theexample of FIG. 1A shows an example of AR navigation on a roadway fromwithin a vehicle, this is merely illustrative, and the AR navigationsystems and methods described herein (e.g., using object masks generatedfrom segmentation methods to blend with an AR path overlay to create anocclusion effect) can be applied to other AR navigation scenarios, suchas walking, driving, indoor, outdoor, etc.

FIGS. 2A-B depicts a flow diagram of an example process for augmentedreality (AR) navigation, in accordance with various aspects of thesubject technology. For explanatory purposes, the example processes ofFIGS. 2A-B is described herein with reference to the components of FIG.1A. Further for explanatory purposes, some blocks of the exampleprocesses of FIGS. 2A-B are described herein as occurring in series, orlinearly. However, multiple blocks of the example processes of FIGS.2A-B may occur in parallel. In addition, the blocks of the exampleprocesses of FIGS. 2A-B need not be performed in the order shown and/orone or more of the blocks of the example processes of FIGS. 2A-B neednot be performed.

In the depicted example flow diagram 200 of FIG. 2A, at block 201, animage such as input image 100 b of FIG. 1A is obtained. The image may beobtained by a camera of a mobile device such as a smart phone, by acamera of a dedicated navigation device, a camera of a vehicle, oranother camera.

At block 202, a guidance indicator (e.g., guidance indicator 116)associated with a navigational guidance route is generated. Generatingthe guidance indicator associated with the navigational guidance routemay include generating an augmented-reality path overlay image such asAR path overlay image 100 d, including the guidance indicator 116. Thenavigational guidance route and/or the guidance indicator may bedetermined or otherwise obtained based on a global positioning systemlocation and map information.

At block 203, the guidance indicator is modified based on at least oneobject (e.g., a roadway object or pathway object such as car 104 on aroadway such as roadway 102) in the image. In order to modify theguidance indicator, a segmentation image such as segmentation image 100c of FIG. 1A may be generated based on the image, the segmentation imagehaving first pixels (e.g., pixels 110) corresponding to the at least oneobject and having a first common value (e.g., one), and second pixels(e.g., pixels 112) having a second common value (e.g., zero). Modifyingthe guidance indicator based on the at least one object in the image mayinclude modifying pixels in the augmented-reality path overlay image 100d that spatially correspond to the first pixels 110 having the firstcommon value in the segmentation image 100 c.

At block 204, the modified guidance indicator (e.g., modified guidanceindicator 117) is provided for display in association with thenavigational guidance route.

In one example, displaying the modified guidance indicator 117 fordisplay in association with the navigational guidance route includesgenerating an output image such as output image 100 f by combining theimage with the augmented-reality path overlay image (e.g., image 100 e)having the modified guidance indicator. In this example, the outputimage may then be displayed with a mobile electronic device or otherdevice.

In another example, displaying the modified guidance indicator fordisplay in association with the navigational guidance route may includeprojecting the augmented-reality path overlay image having the modifiedguidance indicator onto a windshield of a vehicle.

In the depicted example flow diagram 220 of FIG. 2B, at block 221, aninput image such as input image 100 b of FIG. 1A is obtained. The imagemay be obtained by a camera of a mobile device such as a smart phone, bya camera of a dedicated navigation device, a camera of a vehicle, oranother camera.

At block 222, a binary object mask image is generated. The binary objectmasks can be generated based on generating a segmentation image such assegmentation image 100 c of FIG. 1A, the segmentation image having firstpixels (e.g., pixels 110) corresponding to the at least one object andhaving a first common value (e.g., one), and second pixels (e.g., pixels112) having a second common value (e.g., zero).

At block 223, an AR path overlay including a guidance indicator (e.g.,guidance indicator 116) associated with a navigational guidance route isgenerated. Generating the guidance indicator associated with thenavigational guidance route may include generating an augmented-realitypath overlay image such as AR path overlay image 100 d, including theguidance indicator 116. The navigational guidance route and/or theguidance indicator may be determined or otherwise obtained based on aglobal positioning system location and map information.

At block 224, the binary object masks and AR path overlay can be blendedsuch that the guidance indicator is modified based on at least oneobject (e.g., a roadway object or pathway object such as vehicle 104 ona roadway such as roadway 102) in the input image 100 b.

At block 225, the blended binary object masks image and the AR pathoverlay image will form an object mask blended AR path overlay imageincluding a modified guidance indicator. In this example, modifying theguidance indicator is based on the at least a portion of pixels in theAR path overlay image that spatially correspond to the first pixels 110having the first common value in the segmentation image 100 c.

At block 226, the object mask blended AR path overlay image is blendedwith the input image, displaying the modified guidance indicator on theinput image.

At block 227, the blended object mask blended AR path overlay image withthe input image forms an output image and is displayed to the user. Inthis example, the output image may be displayed with a mobile electronicdevice or other device.

FIG. 3 conceptually illustrates electronic system 1000 with which one ormore aspects of the subject technology may be implemented. Electronicsystem 1000, for example, may be, or may be a part of, an augmentedreality (AR) navigation system implemented in standalone device, aportable electronic device such as a laptop computer, a tablet computer,a phone, a wearable device, or a personal digital assistant (PDA), avehicle, or generally any electronic device that transmits signals overa network. Such an electronic system includes various types of computerreadable media and interfaces for various other types of computerreadable media. Electronic system 1000 includes bus 1008, processingunit(s) 1012, system memory 1004, read-only memory (ROM) 1010, permanentstorage device 1002, input device interface 1014, output deviceinterface 1006, and network interface 1016, or subsets and variationsthereof.

In one example, as illustrated in FIGS. 4A and 4B, a guidance indicatorcan be displayed on an image of a driving scene, such as AR path overlayoutput image 400 a. In one example as illustrated in AR path overlayoutput image 400 b, a portion of the guidance indicator, for example aportion of occluded indicator 418, can be occluded from a portion of animage where an object is detected. In this example, the guidanceindicator can have the visual effect of displaying behind the object, orhiding behind or underneath an object when displayed on a display devicefor navigation.

In one example, an input image of a guidance route is obtained. In oneexample, the image obtained can be an image of a physical driving sceneincluding a road 402, one or more moving objects including a vehicle,portions of a vehicle supporting a device configured to capture theinput image, a portion for example including a hood, or dashboard,center console, vehicle side supports, or an objected coupled to anexterior of the vehicle supporting the device, one or more stationaryobjects 407, including a stationary vehicle 406, a sign, a trafficlight, a building, a tree, a curb, a moving pedestrian or stationarypedestrian, other mobile vehicles such as a bicycle, motorcycle,scooter, and other objects that can typically be detected in a drivingscene.

The image may be obtained by a camera of a mobile device such as a smartphone, by a camera of a dedicated navigation device, a camera of avehicle, or another camera. Next, a guidance indicator 416 is generated.The guidance indicator can correspond to the guidance route, theguidance route including a plurality of pixels. In this example, theguidance indicator 416 can include a plurality of arrows pointingtowards the direction of travel to represent the navigation route amoving vehicle associated with the guidance indicator 416 should take.For example, as illustrated in FIG. 4A, a plurality of arrows overlayedonto an image scene of a physical environment represents the guidanceindicator 416 overlayed onto the image of a navigational guidance route.

In one example, a portion of the guidance indicator 416 overlayed ontothe image of the scene may overlap with a portion of the image having anobject. In this example, the brightness or opacity of the portion of theguidance indicator 416 can change such that the brightness or opacity isdifferent from that of the brightness or opacity of the rest of theguidance indicator 416.

In one example, a first object in the path of the guidance indicator isdetected. The first object detected can include a non-moving vehicle, orstationary vehicle 406, or other non-moving automobile. In one example,the first objected detected can be a moving automobile, or other movingvehicle. In one example, the first objected detect can include astationary object detected in the physical environment based on theimage captured by the camera of the mobile device. For example, astationary object can include one of a tree, sign, pedestrian, physicalobstruction in the environment, traffic light, curb, building, bridge,or other stationary objects typically located in a physical urbanenvironment, or a portion or combination of one of a tree, sign,pedestrian, physical obstruction in the environment, traffic light,curb, building, bridge or other stationary objects typically located ina physical urban environment.

In one example, a segmentation image is generated, based on the imagecaptured by the camera. The segmentation image can have one or morepixels corresponding to the first object in the image.

In one example, the guidance indicator 416 can be modified by occludinga portion of the guidance indicator by removing or dimming a portion ofthe plurality of pixels of the guidance indicator, such as occludedindicator 418. In one example, the guidance indicator 416 is modified bymodifying a portion of the guidance indicator based on at least aposition of a first set of pixels in the segmentation image by occludingthe portion of the guidance indicator from first set of pixels. In thisexample, modifying the guidance indicator is based on the at least aportion of pixels in the AR path overlay image that spatially correspondto the first pixels.

As shown, a portion of the guidance indicator 416 is modified, such asoccluded indicator 418, appears to travel beneath a vehicle hood 430 ofthe vehicle supporting the device configured to capture the input imagefor the guidance indicator 416 to overlay.

In one example, the modified guidance indicator can be displayed, as anoutput image, on a display device. In this example, the output image maybe displayed with a mobile electronic device or other device. In oneexample, guidance route, and guidance indicator can be displayed on adisplay device. The display device can be that of the mobile device withthe camera configured to obtain the image, or a separate display devicefrom the mobile device or a separate display device from the camera. Thedisplay device can be operably coupled to a moving vehicle, and whereinthe first object detected is a portion of the moving vehicle. And theportion of the moving vehicle is a hood portion of the vehicle, such asvehicle hood 430, and that occluding of the portion of the guidanceindicator is occluded by the hood portion of the vehicle. In oneexample, other portions of the vehicle can also occlude the guidanceindicator such as a dashboard of a vehicle, physical objects on thedashboard or attached to an exterior of the vehicle, occluding a scenecaptured by the mobile device, a seat, steering wheel, windshield wiper,or a combination thereof, depending on the position of the displaydevice, camera, or both, within the vehicle containing the displaydevice, camera, or both. For example, if the hood is relatively long orlarge such that a larger portion of the image where the guidanceindicator 416 is overlayed contains pixels associated with the vehiclehood 430, or if the camera was placed further back in the vehicle, thenocclusion of the guidance indicator 416 due to the hood, or otherportion of the vehicle, will be greater.

In this example of the occlusion caused by the vehicle hood, an objectmask overlaps to generate occlusion effects on the road are applied withthe same effect on the body of the vehicle supporting the displaydevice, such as a hood or dashboard of the vehicle. Effectively, theguidance indicator 416, during a driving session can display the effectof visually moving beneath the vehicle supporting the display device.Alternatively, a driver or passenger inside the vehicle viewing theoutput image with the modified guidance indicator of the display devicecan effectively view the hood or frontal exterior of the vehicle driveover the guidance indicator 416 mapped and overlayed on the road 402.

In the depicted example flow diagram 500 of FIG. 5, at block 521, aninput image of a guidance route is obtained. The image may be obtainedby a camera of a mobile device such as a smart phone, by a camera of adedicated navigation device, a camera of a vehicle, or another camera.

At block 522, a guidance indicator is generated. The guidance indicatorcan correspond to the guidance route, the guidance route including aplurality of pixels.

At block 523, a first object in the path of the guidance indicator isdetected. The first object detected can include a non-moving vehicle, orother non-moving automobile. In one example, the first objected detectedcan be a moving automobile, or other moving vehicle. In one example, thefirst objected detect can include a stationary object detected in thephysical environment based on the image captured by the camera of themobile device. For example, a stationary object can include one of atree, sign, pedestrian, physical obstruction in the environment, trafficlight, curb, building, bridge, or other stationary objects typicallylocated in a physical urban environment, or a portion or combination ofone of a tree, sign, pedestrian, physical obstruction in theenvironment, traffic light, curb, building, bridge or other stationaryobjects typically located in a physical urban environment.

At block 524, a segmentation image is generated, based on the imagecaptured by the camera. The segmentation image can have one or morepixels corresponding to the first object in the image.

At block 525, the guidance indicator is modified by occluding a portionof the guidance indicator by removing or dimming a portion of theplurality of pixels of the guidance indicator. In one example, theguidance indicator is modified by modifying a portion of the guidanceindicator based on at least a position of a first set of pixels in thesegmentation image by occluding the portion of the guidance indicatorfrom first set of pixels. In this example, modifying the guidanceindicator is based on the at least a portion of pixels in the AR pathoverlay image that spatially correspond to the first pixels.

And at block 526, the modified guidance indicator can be displayed, asan output image, on a display device. In this example, the output imagemay be displayed with a mobile electronic device or other device. In oneexample, guidance route, and guidance indicator can be displayed on adisplay device. The display device can be that of the mobile device withthe camera configured to obtain the image, or a separate display devicefrom the mobile device or a separate display device from the camera. Inthis example, the display device can be operably coupled to a movingvehicle, and wherein the first object detected is a portion of themoving vehicle. And the portion of the moving vehicle is a hood portionof the vehicle such that the occluding of the portion of the guidanceindicator is occluded by the hood portion of the vehicle. In oneexample, other portions of the vehicle can also occlude the guidanceindicator such as a dashboard of a vehicle, physical objects on thedashboard or attached to an exterior of the vehicle, occluding a scenecaptured by the mobile device, a seat, steering wheel, windshield wiper,or a combination thereof, depending on the position of the displaydevice, camera, or both, within the vehicle containing the displaydevice, camera, or both.

Bus 1008 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices ofelectronic system 1000. In one or more embodiments, bus 1008communicatively connects processing unit(s) 1012 with ROM 1010, systemmemory 1004, and permanent storage device 1002. From these variousmemory units, processing unit(s) 1012 retrieves instructions to executeand data to process in order to execute the processes of the subjectdisclosure. The processing unit(s) can be a single processor or amulti-core processor in different embodiments.

ROM 1010 stores static data and instructions that are needed byprocessing unit(s) 1012 and other modules of the electronic system.Permanent storage device 1002, on the other hand, is a read-and-writememory device. This device is a non-volatile memory unit that storesinstructions and data even when electronic system 1000 is off. One ormore embodiments of the subject disclosure use a mass-storage device(such as a magnetic or optical disk and its corresponding disk drive) aspermanent storage device 1002.

Other embodiments use a removable storage device (such as a floppy disk,flash drive, and its corresponding disk drive) as permanent storagedevice 1002. Like permanent storage device 1002, system memory 1004 is aread-and-write memory device. However, unlike storage device 1002,system memory 1004 is a volatile read-and-write memory, such as randomaccess memory. System memory 1004 stores any of the instructions anddata that processing unit(s) 1012 needs at runtime. In one or moreembodiments, the processes of the subject disclosure are stored insystem memory 1004, permanent storage device 1002, and/or ROM 1010. Fromthese various memory units, processing unit(s) 1012 retrievesinstructions to execute and data to process in order to execute theprocesses of one or more embodiments.

Bus 1008 also connects to input and output device interfaces 1014 and1006. Input device interface 1014 enables a user to communicateinformation and select commands to the electronic system. Input devicesused with input device interface 1014 include, for example, alphanumerickeyboards, pointing devices (also called “cursor control devices”),cameras or other imaging sensors, or generally any device that canreceive input. Output device interface 1006 enables, for example, thedisplay of images generated by electronic system 1000. Output devicesused with output device interface 1006 include, for example, printersand display devices, such as a liquid crystal display (LCD), a lightemitting diode (LED) display, an organic light emitting diode (OLED)display, a flexible display, a flat panel display, a solid statedisplay, a projector, or any other device for outputting information.One or more embodiments may include devices that function as both inputand output devices, such as a touch screen. In these embodiments,feedback provided to the user can be any form of sensory feedback, suchas visual feedback, auditory feedback, or tactile feedback; and inputfrom the user can be received in any form, including acoustic, speech,or tactile input.

Finally, as shown in FIG. 3, bus 1008 also couples electronic system1000 to a network (not shown) through network interface 1016. In thismanner, the computer can be a part of a network of computers (such as alocal area network (“LAN”), a wide area network (“WAN”), or an Intranet,or a network of networks, such as the Internet. Any or all components ofelectronic system 1000 can be used in conjunction with the subjectdisclosure.

Many of the above-described features and applications may be implementedas software processes that are specified as a set of instructionsrecorded on a computer readable storage medium (alternatively referredto as computer-readable media, machine-readable media, ormachine-readable storage media). When these instructions are executed byone or more processing unit(s) (e.g., one or more processors, cores ofprocessors, or other processing units), they cause the processingunit(s) to perform the actions indicated in the instructions. Examplesof computer readable media include, but are not limited to, RAM, ROM,read-only compact discs (CD-ROM), recordable compact discs (CD-R),rewritable compact discs (CD-RW), read-only digital versatile discs(e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable/rewritableDVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SDcards, mini-SD cards, micro-SD cards, etc.), magnetic and/or solid statehard drives, ultra-density optical discs, any other optical or magneticmedia, and floppy disks. In one or more embodiments, the computerreadable media does not include carrier waves and electronic signalspassing wirelessly or over wired connections, or any other ephemeralsignals. For example, the computer readable media may be entirelyrestricted to tangible, physical objects that store information in aform that is readable by a computer. In one or more embodiments, thecomputer readable media is non-transitory computer readable media,computer readable storage media, or non-transitory computer readablestorage media.

In one or more embodiments, a computer program product (also known as aprogram, software, software application, script, or code) can be writtenin any form of programming language, including compiled or interpretedlanguages, declarative or procedural languages, and it can be deployedin any form, including as a standalone program or as a module,component, subroutine, object, or other unit suitable for use in acomputing environment. A computer program may, but need not, correspondto a file in a file system. A program can be stored in a portion of afile that holds other programs or data (e.g., one or more scripts storedin a markup language document), in a single file dedicated to theprogram in question, or in multiple coordinated files (e.g., files thatstore one or more modules, sub programs, or portions of code). Acomputer program can be deployed to be executed on one computer or onmultiple computers that are located at one site or distributed acrossmultiple sites and interconnected by a communication network.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, one or more embodiments areperformed by one or more integrated circuits, such as applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In one or more embodiments, such integrated circuits executeinstructions that are stored on the circuit itself.

Those of skill in the art would appreciate that the various illustrativeblocks, modules, elements, components, methods, and algorithms describedherein may be implemented as electronic hardware, computer software, orcombinations of both. To illustrate this interchangeability of hardwareand software, various illustrative blocks, modules, elements,components, methods, and algorithms have been described above generallyin terms of their functionality. Whether such functionality isimplemented as hardware or software depends upon the particularapplication and design constraints imposed on the overall system.Skilled artisans may implement the described functionality in varyingways for each particular application. Various components and blocks maybe arranged differently (e.g., arranged in a different order, orpartitioned in a different way) all without departing from the scope ofthe subject technology.

It is understood that any specific order or hierarchy of blocks in theprocesses disclosed is an illustration of example approaches. Based uponimplementation preferences, it is understood that the specific order orhierarchy of blocks in the processes may be rearranged, or that not allillustrated blocks be performed. Any of the blocks may be performedsimultaneously. In one or more embodiments, multitasking and parallelprocessing may be advantageous. Moreover, the separation of varioussystem components in the embodiments described above should not beunderstood as requiring such separation in all embodiments, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

The subject technology is illustrated, for example, according to variousaspects described above. The present disclosure is provided to enableany person skilled in the art to practice the various aspects describedherein. The disclosure provides various examples of the subjecttechnology, and the subject technology is not limited to these examples.Various modifications to these aspects will be readily apparent to thoseskilled in the art, and the generic principles defined herein may beapplied to other aspects.

A reference to an element in the singular is not intended to mean “oneand only one” unless specifically so stated, but rather “one or more.”Unless specifically stated otherwise, the term “some” refers to one ormore. Pronouns in the masculine (e.g., his) include the feminine andneuter gender (e.g., her and its) and vice versa. Headings andsubheadings, if any, are used for convenience only and do not limit theinvention.

The word “exemplary” is used herein to mean “serving as an example orillustration.” Any aspect or design described herein as “exemplary” isnot necessarily to be construed as preferred or advantageous over otheraspects or designs. In one aspect, various alternative configurationsand operations described herein may be considered to be at leastequivalent.

As used herein, the phrase “at least one of” preceding a series ofitems, with the term “or” to separate any of the items, modifies thelist as a whole, rather than each item of the list. The phrase “at leastone of” does not require selection of at least one item; rather, thephrase allows a meaning that includes at least one of any one of theitems, and/or at least one of any combination of the items, and/or atleast one of each of the items. By way of example, the phrase “at leastone of A, B, or C” may refer to: only A, only B, or only C; or anycombination of A, B, and C.

A phrase such as an “aspect” does not imply that such aspect isessential to the subject technology or that such aspect applies to allconfigurations of the subject technology. A disclosure relating to anaspect may apply to all configurations, or one or more configurations.An aspect may provide one or more examples. A phrase such as an aspectmay refer to one or more aspects and vice versa. A phrase such as an“embodiment” does not imply that such embodiment is essential to thesubject technology or that such embodiment applies to all configurationsof the subject technology. A disclosure relating to an embodiment mayapply to all embodiments, or one or more embodiments. An embodiment mayprovide one or more examples. A phrase such an embodiment may refer toone or more embodiments and vice versa. A phrase such as a“configuration” does not imply that such configuration is essential tothe subject technology or that such configuration applies to allconfigurations of the subject technology. A disclosure relating to aconfiguration may apply to all configurations, or one or moreconfigurations. A configuration may provide one or more examples. Aphrase such a configuration may refer to one or more configurations andvice versa.

In one aspect, unless otherwise stated, all measurements, values,ratings, positions, magnitudes, sizes, and other specifications that areset forth in this specification, including in the claims that follow,are approximate, not exact. In one aspect, they are intended to have areasonable range that is consistent with the functions to which theyrelate and with what is customary in the art to which they pertain.

It is understood that some or all steps, operations, or processes may beperformed automatically, without the intervention of a user. Methodclaims may be provided to present elements of the various steps,operations or processes in a sample order, and are not meant to belimited to the specific order or hierarchy presented.

All structural and functional equivalents to the elements of the variousaspects described throughout this disclosure that are known or latercome to be known to those of ordinary skill in the art are expresslyincorporated herein by reference and are intended to be encompassed bythe appended claims. Moreover, nothing disclosed herein is intended tobe dedicated to the public regardless of whether such disclosure isexplicitly recited in the claims. No claims element is to be construedunder the provisions of 35 U.S.C. § 112 (f) unless the element isexpressly recited using the phrase “means for” or, in the case of amethod, the element is recited using the phrase “step for.” Furthermore,to the extent that the term “include,” “have,” or the like is used, suchterm is intended to be inclusive in a manner similar to the term“comprise” as “comprise” is interpreted when employed as a transitionalword in a claim.

The Title, Background, Brief Description of the Drawings, and Claims ofthe disclosure are hereby incorporated into the disclosure and areprovided as illustrative examples of the disclosure, not as restrictivedescriptions. It is submitted with the understanding that they will notbe used to limit the scope or meaning of the claims. In addition, in theDetailed Description, it can be seen that the description providesillustrative examples and the various features are grouped together invarious embodiments for the purpose of streamlining the disclosure. Thismethod of disclosure is not to be interpreted as reflecting an intentionthat the claimed subject matter requires more features than areexpressly recited in any claim. Rather, as the following claims sreflect, inventive subject matter lies in less than all features of asingle disclosed configuration or operation. The following claims arehereby incorporated into the Detailed Description, with each claimsstanding on its own to represent separately claimed subject matter.

The claims are not intended to be limited to the aspects describedherein, but are to be accorded the full scope consistent with thelanguage of the claims and to encompass all legal equivalents.Notwithstanding, none of the claims are intended to embrace subjectmatter that fails to satisfy the requirement of 35 U.S.C. § 101, 102, or103, nor should they be interpreted in such a way.

The invention claimed is:
 1. A method, comprising: generating a guidanceindicator corresponding to a guidance route; obtaining an image of theguidance route including an object; generating a segmentation imagebased on the image, the segmentation image having first pixelscorresponding to the object in the image; modifying a portion of theguidance indicator based on at least a position of the first pixels inthe segmentation image by occluding the portion of the guidanceindicator from first pixels; and displaying the modified guidanceindicator on a display.
 2. The method of claim 1, wherein generating thesegmentation image is performed by applying semantic segmentation on theimage.
 3. The method of claim 2, wherein the image comprises an image ofa roadway, and wherein the at least one object comprises at least oneobject on the roadway.
 4. The method of claim 3, wherein generating theguidance indicator associated with the navigational guidance routecomprises generating an augmented-reality path overlay image includingthe guidance indicator.
 5. The method of claim 4, wherein modifying theguidance indicator based on the at least one object in the imagecomprises modifying pixels in the augmented-reality path overlay imagethat spatially correspond to the first pixels in the segmentation image.6. The method of claim 5, wherein displaying the modified guidanceindicator in association with the navigational guidance route comprisesgenerating an output image by combining the image with theaugmented-reality path overlay image having the modified guidanceindicator.
 7. The method of claim 6, further comprising displaying theoutput image on a display of a mobile electronic device.
 8. The methodof claim 5, wherein displaying the modified guidance indicator inassociation with the navigational guidance route comprises projectingthe augmented-reality path overlay image having the modified guidanceindicator onto a windshield of a vehicle.
 9. The method of claim 1,further comprising determining the navigational guidance route based ona global positioning system location and map information.
 10. Anavigation system comprising one or more non-transitorycomputer-readable media storing computer-executable instructions that,when executed on one or more processors, cause the one or moreprocessors to perform acts comprising: generating a guidance indicatorcorresponding to a guidance route; obtaining an image of the guidanceroute including an object; generating a segmentation image based on theimage, the segmentation image having first pixels corresponding to theobject in the image; modifying a portion of the guidance indicator basedon at least a position of the first pixels in the segmentation image;and displaying the modified guidance indicator on a display.
 11. Thenavigation system of claim 10, wherein generating the segmentation imageis performed by applying semantic segmentation on the image.
 12. Thenavigation system of claim 11, wherein the image comprises an image of aroadway, and wherein the at least one object comprises at least oneobject on the roadway.
 13. The navigation system of claim 12, whereingenerating the guidance indicator associated with the navigationalguidance route comprises generating an augmented-reality path overlayimage including the guidance indicator.
 14. The navigation system ofclaim 13, wherein modifying the guidance indicator based on the at leastone object in the image comprises modifying pixels in theaugmented-reality path overlay image that spatially correspond to thefirst pixels in the segmentation image.
 15. The navigation system ofclaim 14, wherein displaying the modified guidance indicator inassociation with the navigational guidance route comprises generating anoutput image by combining the image with the augmented-reality pathoverlay image having the modified guidance indicator.
 16. The navigationsystem of claim 15, further comprising instructions for displaying theoutput image on a display of a mobile electronic device.
 17. Thenavigation system of claim 16, wherein displaying the modified guidanceindicator in association with the navigational guidance route comprisesprojecting the augmented-reality path overlay image having the modifiedguidance indicator onto a windshield of a vehicle.
 18. A method forgenerating a path overlay, the method comprising: obtaining an image ofthe guidance route; generating a guidance indicator corresponding to aguidance route, the guidance indicator comprising a plurality of pixels;detecting a first object in the path of the guidance indicator;generating a segmentation image based on the image, the segmentationimage having first pixels corresponding to the first object in theimage; modifying the guidance indicator by occluding a portion of theguidance indicator by removing or dimming a portion of the plurality ofpixels of the guidance indicator; and displaying the modified guidanceindicator on a display.
 19. The method of claim 18, wherein the firstobject detected is a non-moving vehicle.
 20. The method of claim 18,wherein the first object detected is a stationary object in theenvironment.
 21. The method of claim 20, wherein the stationary objectis one of a tree, sign, pedestrian, physical obstruction in theenvironment, traffic light, curb, building, bridge.
 22. The method ofclaim 18, wherein the display is operably coupled to a moving vehicle,and wherein the first object detected is a portion of the movingvehicle.
 23. The method of claim 22, wherein the portion of the movingvehicle is a hood portion of the vehicle such that the occluding of theportion of the guidance indicator is occluded by the hood portion of thevehicle.